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Introduction 


Children  with  autism  are  being  diagnosed  at  increasing  numbers  (Center  for  Disease 
Control  suggests  1  in  150).  This  has  placed  a  demand  to  conduct  an  appropriate 
diagnostic  work-up  which  may  be  difficult  in  rural,  underserved  areas.  A  potential 
solution  used  in  other  areas  of  medicine  for  specialty  care  has  been  telemedicine.  We 
explored  the  use  of  telemedicine  in  the  diagnosis  of  autism  to  determine  if  autism  could 
be  accurately  diagnosed,  families  were  satisfied  with  the  diagnostic  process,  and  gold 
standards  for  the  assessment  of  autism  (Autism  Diagnostic  Observation  Schedule  and 
Autism  Diagnostic  Interview-Revised)  were  equally  reliable  in  a  telemedicine  versus  a 
live  condition.  Twenty-one  children  between  the  ages  of  three  and  five  were  recruited. 
Half  of  the  children  had  an  existing  diagnosis  of  autism  and  half  developmental  delay. 
These  children  were  randomly  assigned  to  either  a  telemedicine  or  live  assessment 
condition.  Results  indicated  that  clinicians  were  equally  accurate  in  their  diagnosis, 
families  were  satisfied,  and  reliability  measures  on  gold  standards  were  no  different  in 
the  live  versus  the  telemedicine  condition.  Future  research  includes  a  prospective  study 
comparing  telemedicine  to  live  assessments  with  more  and  a  broader  range  of  subjects. 

Body 

Children  with  autism  are  being  screened  and  diagnosed  at  alarming  rates.  Current  data 
from  the  Center  for  Disease  Control  indicates  that  1  of  150  children  have  autism.  These 
rates  have  placed  an  increased  demand  on  providers  to  conduct  an  appropriate 
diagnostic  workup.  The  need  for  providers  who  can  conduct  an  assessment  for  autism 
is  particularly  evident  in  rural  communities.  One  possible  solution  to  this  problem  is  the 
use  of  telemedicine.  Telemedicine  is  defined  as  the  use  of  real-time  video  conferencing 
for  clinical  purposes.  Telemedicine  allows  children  and  their  families  in  rural 
communities  access  to  services  without  time  consuming  and  costly  travel. 
Telepsychiatry  has  been  found  to  reduce  costs  for  patients  related  to  travel  (Hyler  & 
Gangure,  2003).  The  estimated  savings  for  general  telemedicine  including  travel 
expenses,  work  hours  lost  and  hospital  visits  avoided  was  as  high  as  $224  per  visit 
(Young  &  Ireson,  2003).  Telemedicine  has  been  suggested  for  specialty  care 
consultation  for  children  with  special  health  care  needs  (Karp,  et  al.  1999).  This  study 
found  that  most  telemedicine  involved  pediatric  allergy  (35%),  pulmonology  (29%), 
neurology  (19%)  and  genetics  (16%).  Both  physicians  and  family  members  were 
generally  positive  about  telemedicine  as  a  way  of  improving  collaboration 

In  the  January  2004  issue  of  Pediatrics,  Marcin  et  al.  presented  evidence  that  Children 
with  Special  Healthcare  Needs  including  autism  could  be  provided  with  medical  care  in 
underserved  rural  communities  for  their  specially  needs  using  telemedicine  and  that 
families  were  satisfied  with  these  services. 

Nesbitt  et  al.  (2006)  developed  guidelines  for  effective  telemedicine  technologies  and 
delivering  empirically  supported  services  for  children  with  autism  living  in  rural 
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communities  using  telemedicine  technology.  This  project  highlighted  the  need  for 
effective  strategies  to  deliver  services  for  those  with  autism,  when  families  live  in 
underserved  areas  and  suggested  research  is  needed  to  evaluate  clinical  services 
delivered  over  telemedicine. 

Practice  parameters  for  screening  and  diagnosis  of  autism  from  the  American 
Academy  of  Neurology  indicate  that  adequate  diagnosis  includes  a  parent  interview  for 
autism  such  as  the  Autism  Diagnostic  Interview  (ADI)  and  a  structured  observation  of 
the  child  using  validated  instruments  such  as  the  Autism  Diagnostic  Observation 
Schedule  (ADOS).  The  ADI  and  ADOS  are  currently  gold  standards  that  are  used  to 
diagnose  autism.  These  measures  along  with  a  medical  history,  review  of  records,  and 
clinic  observations  are  used  in  an  autism  assessment  to  determine  if  the  child  meets 
the  DSM-IV  criteria  for  autism. 

The  proposed  research  questions  included:  Is  there  a  difference  in  reliability  of  the  ADI 
and  ADOS  when  observers  are  watching  live  versus  over  telemedicine?,  Is  there  a 
difference  in  diagnostic  impressions  regarding  autism  when  observers  are  evaluating 
live  versus  telemedicine?,  and  is  there  a  difference  in  satisfaction  between  onsite  and 
telemedicine  evaluations  of  autism?. 

Statement  of  Work  (SOW) 

Goal  1 -Recruitment  of  children 
May  2008-July  2008: 

Project  Coordinator,  Rene  Jamison,  PhD.  and  Project  Director,  Ronald  Matthew  Reese, 
PhD.  will  revise  the  Human  Subjects  Protocol  for  the  study  and  resubmit  to  the  Human 
Research  Committee  (HRC). 

Human  Research  Committee  approval  was  granted  on  June  4,  2008. 

May  2008-March  2009: 

Between  May  2008  and  March  2009,  Dr.  Jamison  will  meet  with  Dr.  Reese  to  identify 
subjects  who  might  be  recruited  for  the  study.  The  subjects  will  be  identified  from  the 
clinics  where  the  study  will  take  place.  After  HSC  approval,  Dr.  Jamison  and  the 
research  assistant  will  call  potential  candidates,  obtain  consent,  and  schedule  them  into 
the  study.  Two-three  subjects  will  be  recruited  per  month. 

In  order  to  ensure  participants  met  the  inclusion  criteria  (either  diagnosis  of  an  autism 
spectrum  disorder  or  developmental  delay),  participants  were  only  recruited  if  evaluated 
at  our  clinic.  Therefore  recruitment  efforts  focused  on  reviewing  patient  charts  from  the 
past  year,  identifying  children  who  were  between  the  ages  of  3-5  years  old  and  living  in 
the  Kansas  City  or  surrounding  area.  Recruitment  efforts  also  focused  on  newly 
diagnosed  patients  by  providing  clinicians  as  well  as  families  information  about 
opportunities  for  study  participation.  In  August  of  2008  a  recruitment  flyer  was 
developed  and  distributed  in  clinic  areas  and  distributed  at  community  events 
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participated  in  by  our  center.  Dr.  Jamison  (Project  Director)  and  Josh  Turek  (research 
assistant)  reviewed  patient  reports  from  the  birth  to  five  year  old  clinics  to  identify 
potential  participants  for  the  study.  Josh  Turek  contacted  potential  participants  and 
scheduled  participants  for  their  study  visit.  To  date,  a  total  of  21  participants  have  been 
recruited  and  completed  their  study  visit.  Eleven  of  these  participants  were  previously 
diagnosed  with  an  autism  spectrum  disorder,  while  10  were  previously  diagnosed  with 
developmental  delays.  Some  of  the  children  appeared  to  change  diagnosis  since  their 
first  diagnostic  visit.  All  of  the  children  had  been  referred  because  of  red  flags  for  autism 
so  it  is  understandable  the  diagnosis  may  have  changed  in  some  cases.  The  pilot  data 
will  give  us  information  for  a  prospective  study  with  sufficient  power  to  determine  if  there 
is  a  difference  between  telemedicine  and  live  visits.  Also  the  fact  that 
children  were  already  diagnosed  may  have  affected  parents’  perception  of  the 
telemedicine  diagnostic  process. 

Goal  2:  Testing  of  Children 
June  2008-March  2009: 

Researchers  Dr.  Maura  Wendland,  Matthew  Braun,  SLP,  Dr.  Jamison,  and  Dr.  Reese 
will  evaluate  subjects  using  the  ADOS,  ADI-R,  and  medical  interview.  Two  of  these 
research  personnel  will  be  with  the  parents  and  child  in  the  same  room  and  two  will 
evaluate  through  telemedicine.  For  half  the  families  the  assessment  will  be  directed  by 
the  research  personnel  who  are  in  the  room  and  half  will  be  evaluated  with  the  research 
personnel  directing  the  assessment  over  telemedicine.  All  research  personnel  have 
been  trained  in  the  ADOS,  ADI,  and  the  diagnostic  criteria  for  autism.  Research 
personnel  will  be  randomly  assigned  to  live  versus  telemedicine  conditions  and  will  not 
have  seen  the  subjects  in  the  study. 

All  members  of  the  research  were  fluent  in  the  administration  and  scoring  of  both  the 
ADOS  and  ADI-R  measures.  Four  of  the  five  research  team  members  had  completed 
ADOS  clinical  user  training  and  the  PI  has  also  completed  the  ADOS  research  training, 
prior  to  beginning  the  study.  All  members  of  the  research  team  completed  practice 
scoring  tapes,  with  independent  scoring  followed  by  group  discussions  to  help  work 
towards  reliability  on  ADOS  scoring.  The  team  completed  a  total  of  5  practice  scoring 
sessions,  with  2  of  3  consecutive  sessions  at  72%  or  more  interobserver  agreement. 
Participants  were  randomly  assigned  to  condition  (telemedicine  or  live),  with  half  of  the 
participants  in  the  autism  group  in  each  condition  (5  live,  5  telemedicine)  and  half  of  the 
participants  in  the  developmental  delay  group  in  each  condition  (5  live,  5  telemedicine). 
In  the  telemedicine  condition,  the  PI,  Dr.  Reese,  directed  families  to  complete  ADOS 
activities  with  their  child  over  interactive  television  and  completed  the  ADI-R  interview 
over  interactive  television.  In  the  live  condition,  the  PI,  Dr.  Reese,  directed  families  to 
complete  ADOS  activities  with  their  child  while  in  the  same  room  with  the  family  and 
completed  the  ADI-R  interview  in  the  room  as  well.  Clinicians  were  randomly  assigned 
to  create  pairs,  with  two  clinicians  in  each  setting  (telemedicine  or  live)  for  each 
participant.  In  order  to  limit  administration  variability,  Dr.  Reese  directed  all  families, 
regardless  of  condition.  Assessments  were  independently  scored  by  the  four  clinicians. 
Dr.  Reese,  in  addition  to  directing  the  assessment,  was  also  responsible  for  scoring  the 


assessments  as  part  of  the  four  person  team.  Following  ADOS  activities  and  ADI-R 
interview,  caregivers  completed  a  brief  satisfaction  survey  and  was  provided 
compensation  for  their  participation  ($30.00  gift  card  to  target).  Brief  feedback  was  also 
provided  for  the  caregivers,  including  comments  on  the  child’s  strengths,  areas  to 
develop  and  future  treatment  directions  for  the  child  and  family. 

Goal  3:  Scoring  of  Tests  and  Assigning  a  Diagnosis 
June  2008-March  2009 

All  four  researchers  will  independently  score  the  ADOS  and  ADI  and  will  assign  a  DSM- 
IV  Diagnosis.  Subjects  will  be  assigned  a  number  by  Dr.  Jamison  and  the  research 
assistant  so  that  they  cannot  be  identified.  Protocols  and  diagnostic  sheets  will  be  kept 
in  a  locked  cabinet  and  supervised  by  Dr.  Jamison.  Identifiers  will  be  kept  in  a  separate 
locked  cabinet.  Dr.  Jamison  and  the  research  assistant  will  be  responsible  for  entering 
data  for  each  subject  on  a  spreadsheet. 

Each  ADOS  and  ADI  was  independently  scored  by  4  of  5  researchers.  Upon 
completion,  the  research  assistant  entered  the  data  into  a  SPSS  database  developed  in 
December  2008.  All  research  participants  and  their  families  were  assigned  a  number 
(1-21).  A  master  list  was  created  and  kept  in  a  separate  file  that  contained  participant 
identifying  information  along  with  their  participant  ID  number.  A  file  was  created  for 
each  participant  containing  unidentifiable  raw  data,  with  the  completed  ADOS  protocols, 
completed  ADI  protocols,  and  completed  family  satisfaction  survey  and  feedback  forms 
maintained  in  each  file.  Signed  consents  were  kept  in  a  separate  file  with  no  subject 
number  attached.  All  participant  folders  were  maintained  in  a  locked  file  cabinet  in  an 
office  with  a  secure,  keyless  entry  for  participant  security. 

Study  information  for  all  participants  was  entered  into  an  SPSS  data  base  developed  by 
Dr.  Jamison  and  Josh  Turek.  Additional  consultation  was  provided  by  Kandace 
Fleming  regarding  data  base  development  and  the  creation  of  appropriate  variables  for 
future  analysis.  Data  was  entered  in  a  timely  manner,  typically  within  a  week  of 
completion  of  assessments. 

Goal  4:  Measuring  Parent  Satisfaction 
May  2008-June  2008 

Dr.  Jamison  and  Dr.  Reese  will  develop  parent  satisfaction  evaluation  from  existing 
telemedicine  forms. 

A  satisfaction  survey  was  revised  based  on  existing  telemedicine  satisfaction  forms  in 
July  of  2008.  Family  satisfaction  was  measured  using  a  7  point  Likert-type  scale, 
ranging  from  1  (Strongly  Disagree)  to  7  (Strongly  Agree).  See  addendum  1 .  Items  4,  5, 
8-10  were  reverse  scored,  with  1  indicating  positive  feelings  and  7  indicating  negative 
feelings.  Questions  1-7  were  completed  by  all  participants  while  questions  8-13  were 
only  completed  by  participants  in  the  telemedicine  condition. 


June  2008-March  2009 
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Dr.  Jamison  and  the  research  assistant  will  administer  the  parent  satisfaction  form,  file 
forms,  and  enter  data. 

Satisfaction  forms  were  collected  and  entered  into  the  database  for  all  participants,  with 
study  visits  from  January  2009  to  September  2009.  All  21  caregivers  of  participants 
completed  a  satisfaction  survey  following  study  measures.  Partial  data  is  missing  from 
two  participants,  each  failing  to  answer  two  questions,  which  seemed  to  be  a  random 
error.  In  general,  families  in  both  the  Live  and  Telemedicine  conditions  rated  their 
satisfaction  with  the  experience  as  high  (see  summary  of  results  below). 

Goal  5:  Data  Analysis,  Write  Up,  and  Presentation 
April  2009-May  2009 

Dr.  Jamison  and  Dr.  Reese  will  analyze  the  data  to  determine  if  there  are  significant 
differences  between  the  ADOS  and  ADI  reliability  scores,  diagnostic  impressions,  and 
parent  satisfaction  when  the  assessments  are  completed  live  as  compared  to 
telemedicine.  This  data  will  be  presented  at  autism  and  telemedicine  conferences  and 
written  up  for  publication. 

A  preliminary  analysis  of  the  data  was  conducted  in  May  of  2009  for  presentation  at  our 
annual  LEND  trainee  research  forum,  with  data  presented  by  Josh  Turek,  research 
assistant.  Final  data  analysis  was  conducted  in  September  of  2009  by  the  research 
team  and  statistical  consultant.  We  evaluated  the  following  hypotheses: 

1 .  We  expected  that  the  ADOS  and  ADI  will  be  as  reliable,  based  on  inter-rater 
agreement  at  the  item  level,  when  scored  live  versus  telemedicine. 

2.  Families  will  be  equally  satisfied  (according  to  satisfaction  survey)  with  their 
evaluation  experience,  regardless  of  evaluation  setting  (live  versus 
telemedicine). 

3.  Clinicians  will  be  equally  reliable  identifying  the  diagnosis  regardless  of  condition 
(live  versus  telemedicine). 

In  order  to  determine  if  percent  agreement  for  the  ADOS  items  were  as  high  in  the 
telemedicine  condition  as  they  were  in  the  live  condition,  the  individual  item  ratings  for 
each  of  the  four  raters  were  compared  in  sets.  The  scores  for  the  two  live  raters  and 
two  telemedicine  raters  were  compared.  If  the  item  score  for  the  two  raters  was 
identical,  then  an  agreement  was  scored.  If  the  scores  were  not  identical,  then  a 
disagreement  was  scored.  Similar  scores  were  calculated  for  both  the  two  telemedicine 
raters  and  two  live  raters.  These  agreement/disagreement  scores  were  then 
summarized  across  the  21  clients  to  yield  a  percent  agreement  score  for  each  condition 
by  item.  Occasionally  an  item  was  missing  for  an  individual  child,  so  percentages  are 
not  necessarily  consistent  across  items.  Percent  agreement  scores  for  each  of  the 
ADOS  items  are  presented  in  Table  1  for  each  condition.  A  difference  score  was 
calculated  for  each  item  in  order  to  summarize  the  magnitude  of  the  discrepancy 
between  agreement  percentages  and  the  direction  of  the  higher  score  across  items. 

For  14  items,  percent  agreement  was  higher  in  the  Live  condition,  although  many  of  the 
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differences  were  quite  small.  For  1 3  items,  percent  agreement  was  higher  in  the 
Telemedicine  condition.  For  two  of  the  items  there  was  no  difference  in  percent  of 
agreement.  The  magnitude  of  the  difference  was  similar  across  conditions.  For  items 
where  agreement  was  higher  in  the  Live  condition,  the  difference  was  13  percentage 
points  higher  on  average.  For  items  in  which  Telemedicine  yielded  higher  agreement, 
telemedicine  was  1 1  percentage  points  higher  on  average.  Contingency  tables  with  a 
dichotomous  variable  for  Live/Telemedicine  and  a  second  variable  coded  as 
agreement/disagreement  were  evaluated  for  each  of  the  ADOS  items.  The  Chi  Square 
calculated  was  significant  at  the  p=.05  level  for  only  two  items  (ADOS  A7  and  ADOS 
El ).  Controlling  the  Type  I  error  rate  for  the  number  of  tests,  only  the  ADOS  A7  Chi 
Square  would  be  considered  to  be  significant  (X2(1)  =13.10,  p<.001).  Item  A7  is  related 
to  socially  directed  pointing,  with  results  suggesting  raters  agreed  more  often  in  the  Live 
condition  (inter-rater  agreement  for  Live  =  86%  versus  Telemedicine  =  30%). 

Table  2  provides  the  percent  agreement  values  for  the  ADI  items  by  condition  as  well 
as  the  difference  between  conditions.  The  differences  in  agreement  between  Live  and 
Telemedicine  were  generally  smaller  for  the  ADI  items  than  for  the  ADOS  items.  For  1 0 
of  the  ADI  items,  percent  agreement  was  higher  in  the  live  condition  than  in  the 
telemedicine  condition  (the  average  difference  was  7%  for  these  items).  For  1 8  of  the 
ADI  items,  the  percent  agreement  was  higher  in  the  Telemedicine  condition  than  in  the 
Live  condition  (11%  higher  on  average).  For  8  of  the  ADI  items  there  was  no  difference 
in  percent  agreement.  The  Chi  Square  Tests  indicated  that  only  one  of  the  items  was 
significantly  different  in  agreement  across  Live  and  Telemedicine,  item  71  (X2(1)  =5.56, 
p=.018).  Item  71  is  related  to  unusual  sensory  interests,  with  results  suggesting  raters 
agreed  more  often  in  the  Telemedicine  condition. 

In  order  to  determine  if  families  who  were  evaluated  in  the  Telemedicine  setting  were 
equally  as  satisfied  with  the  evaluation  as  families  who  were  evaluated  in  the  Live 
setting,  parent’s  responses  to  seven  items  about  the  quality  of  the  evaluation 
experience  were  examined.  Parents  responded  to  each  of  the  seven  items  using  a 
Likert  scale  which  ranged  from  1  (Strongly  Disagree)  to  7  (Strongly  Agree).  Two  of  the 
items  were  reversed  scored  so  that  high  scores  would  indicate  high  satisfaction  with  the 
evaluation  for  all  items.  Mean  scores  were  calculated  for  each  family  averaging  scores 
across  the  seven  possible  items.  If  a  family  answered  at  least  6  of  the  7  items,  a  mean 
across  all  available  items  was  calculated.  If  a  family  answered  less  than  6  items,  then 
the  family  was  missing  a  mean  satisfaction  score.  Independent  Samples  t-tests  were 
calculated  for  each  of  the  items  and  the  mean  across  all  seven  items  to  determine  if  the 
mean  satisfaction  was  significantly  different  based  on  condition.  Table  3  presents  the 
means,  standard  deviations,  t-test  summary,  and  effect  size  (Cohen’s  d)  for  each 
question  individually  and  for  the  mean  satisfaction  across  all  seven  items.  None  of  the 
probability  values  for  the  t-tests  would  indicate  that  there  are  significant  differences  in 
mean  satisfaction  between  the  Live  and  Telemedicine  families  as  all  of  the  p-values 
were  greater  than  .15.  Because  of  the  relatively  small  sample  sizes  within  each 
condition,  effect  sizes  are  also  important  to  consider  when  drawing  conclusions  about 
these  data.  Two  of  the  effect  sizes  for  the  satisfaction  data  are  moderate  suggesting 
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that  further  examination  of  satisfaction  with  a  larger  sample  is  warranted.  Item  7  which 
asks  families  to  rate  their  overall  satisfaction  with  the  evaluation  visit  was  uniformly 
rated  as  highly  as  possible  by  the  telemedicine  families  yielding  a  mean  of  7.0.  The 
overall  rating  was  also  high  for  the  Live  families,  but  the  effect  size  was  moderate 
suggesting  a  potentially  meaningful  difference  in  satisfaction  favoring  the  Telemedicine 
families.  Conversely,  the  effect  size  for  average  satisfaction  across  all  seven  items  was 
also  moderate  suggesting  that  average  satisfaction  in  the  Live  condition  was  higher 
than  satisfaction  in  the  Telemedicine  condition.  This  discrepancy  in  finding  may 
indicate  that  there  is  really  not  a  meaningful  difference  in  satisfaction  across  condition 
or  it  may  be  that  with  larger  sample  sizes  the  potential  differences  would  be  clarified.  A 
larger  study  is  being  planned  to  further  examine  satisfaction  with  evaluation  across  Live 
and  Telemedicine  conditions.  Given  an  effect  size  (d)  of  .6  as  observed  in  this  study, 
one  would  need  to  have  a  sample  of  45  per  group  or  90  total  to  detect  a  significant 
difference  between  groups  at  .80  power. 

In  order  to  determine  if  clinicians  were  equally  reliable  identifying  diagnosis  regardless 
of  condition  (Live  vs.  Telemedicine),  each  of  the  four  raters  who  were  observing  the 
evaluation  (2  Live  and  2  Telemedicine)  were  asked  to  indicate  whether  the  observed 
child  should  have  a  diagnosis  of  DD  or  ASD  based  on  their  professional  observations 
during  the  session  and  scores  from  the  autism  assessments.  These  predicted 
diagnoses  were  then  compared  to  the  actual  previous  diagnosis  given  to  the  family. 

The  Live  and  Telemedicine  clinicians  were  extremely  consistent  in  their  assignment  of 
diagnoses  and  only  differed  on  the  rating  for  a  single  child  when  one  of  the  Live  coders 
differed  in  assigned  diagnosis  from  the  other  Live  coder  and  the  two  Telemedicine 
coders.  Thus,  the  percentage  agreement  with  previous  diagnosis  was  not  significantly 
different  across  condition  with  83%  of  the  diagnosis  ratings  made  by  Live  raters  and 
86%  of  the  diagnosis  ratings  made  over  Telemedicine  agreeing  with  previous 
diagnosis.  It  is  possible  that  the  correct  diagnosis  for  some  of  the  children  was  different 
from  their  previous  diagnosis  and  thus  the  agreement  between  previous  diagnosis  and 
diagnosis  during  the  evaluation  is  not  closer  to  100%. 

The  data  will  be  presented  at  the  Kansas  Center  for  Autism  Research  and  Training  (K- 
CART)  autism  conference  in  November  in  Wichita,  Kansas.  We  are  currently  writing  a 
manuscript  and  plan  to  submit  the  manuscript  by  the  end  of  December  to  the  Journal  of 
Autism  and  Developmental  Disabilities,  Focus  on  Autism  and  Developmental  Disorders, 
Autism,  or  Journal  of  the  Association  of  Intellectual  and  Developmental  Disabilities. 

Goal  6:  Grant  Development 
May  2009-August  2009 

Telemedicine  grants  will  be  developed  using  the  innovative  grant  as  pilot  data. 

We  submitted  an  R40  grant  to  the  Maternal  Child  and  Health  Bureau  (MCHB)  in 
response  to  their  Intervention  in  Autism  competition  and  proposed  to  compare 
telemedicine,  with  outreach  and  hospital  based  clinics  in  terms  of  cost  effectiveness 
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and  access  to  empirically  based  treatments  after  the  visit.  We  used  preliminary  data 
from  this  DOD  grant  (telemedicine  data)  as  pilot  data  in  an  attempt  to  secure  funding  for 
this  future  project.  The  grant  made  it  to  the  review  panel  and  was  scored.  The 
reviewers  were  impressed  with  the  pilot  data  but  criticized  the  grant  for  the  lack  of 
random  assignment  to  conditions  although  were  using  a  co-variable  to  control  for 
differences  between  conditions.  We  will  resubmit  to  a  similar,  revised  proposal  to 
MCHB,  NIH  and/or  Autism  Speaks. 

We  now  have  pilot  data  for  a  larger  prospective  study  with  sufficient  power  to  determine 
if  there  are  differences  in  inter-rater  agreement  on  the  autism  tools  when  scored  live 
versus  over  telemedicine.  These  data  were  accepted  for  a  presentation  at  the  K-CART 
conference  and  also  discussed  at  the  Kansas  Children's  Cabinet  meeting  for  financial 
support.  The  Children's  Cabinet  awarded  $50,000  to  help  us  serve  families  who  had 
children  suspected  of  autism  and  live  in  rural  communities  and  continue  our  research 
on  family  satisfaction  with  telemedicine  services  in  autism.  Our  colleagues  also 
received  a  NIDRR  grant  to  examine  telemedicine  as  an  approach  for  parent  training  to 
treat  children  with  autism. 

We  continue  to  collaborate  with  the  Department  of  Telemedicine  at  The  University  of 
Kansas  Medical  Center  which  is  one  of  the  top  telemedicine  departments  in  the  nation 
and  has  received  national  awards 


Key  Research  Accomplishments 

•  Paper  presentation  at  Association  of  University  Centers  on  Disability  national 
meeting  in  November  2009  in  Washington,  D.C. 

•  Preliminary  data  presented  at  the  Kansas  Children's  Cabinet  meeting  and  at  our 
annual  LEND  trainee  research  forum. 

•  Data  to  be  presented  at  the  KCART  autism  conference  in  November  2009. 

•  Preliminary  date  utilized  in  development  of  a  MCHB  Intervention  in  Autism 
competition  grant  application. 

•  There  have  been  no  published  studies  examining  the  use  of  telemedicine  to  assess 
and  treat  autism 

•  We  are  currently  writing  a  manuscript  and  plan  to  submit  the  manuscript  by  the  end 
of  December  to  the  Journal  of  Autism  and  Developmental  Disabilities,  Focus  on 
Autism  and  Developmental  Disorders,  Autism,  or  Journal  of  the  Association  of 
Intellectual  and  Developmental  Disabilities. 

•  Members  of  this  research  team  submitted  a  grant  to  Autism  Speaks  continuing 
research  in  telemedicine,  focusing  on  intervention  for  families  with  children  with 
autism  (The  Delivery  of  Parent-Child  Interaction  Therapy  over  Interactive 
Television:  Outcomes  for  Families  of  Children  with  Autism). 


Reportable  Outcomes 
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•  Overall  levels  of  consumer  satisfaction  indicate  high  satisfaction  with  telemedicine 
as  a  service  delivery  model  and  we  have  expanded  our  clinics  based  on  our  data 

•  Differences  between  the  clinicians  scoring  via  telemedicine  versus  live  were  non¬ 
significant,  indicating  telemedicine  as  a  viable  option  for  use  of  autism  tools  and 
diagnosis  for  families  with  minimal  access  to  specialty  services  in  their  region. 

•  The  differences  in  diagnostic  conclusion  reached  were  non-significant,  indicating 
reliability  between  conditions. 


Conclusion 

While  current  data  indicates  approximately  1  of  150  children  have  autism,  these  rates 
have  placed  an  increased  demand  on  providers  to  conduct  an  appropriate  diagnostic 
assessment.  The  need  for  providers  who  can  conduct  an  assessment  for  autism  is 
particularly  evident  in  rural  communities.  Advances  in  technology  have  aided  the 
possible  solution  to  this  problem  in  the  use  of  interactive  television,  telemedicine,  as  a 
means  to  provide  services  to  families  near  their  home  community.  In  order  to  ensure 
accurate  diagnosis  and  evaluation  procedures,  it  is  important  to  evaluate  the  “gold 
standard”  tools  used  to  aid  diagnosis  of  autism  for  use  with  this  new  technology.  The 
current  pilot  study  evaluated  the  ADOS  and  ADI-R  for  use  in  telemedicine  to  begin 
evaluating  the  accuracy  and  feasibility  of  using  telemedicine  as  a  viable  means  to 
diagnose  autism  spectrum  disorders.  The  main  research  questions  included:  Is  there  a 
difference  in  reliability  of  the  ADI  and  ADOS  when  observers  are  watching  live  versus 
over  telemedicine?,  Is  there  a  difference  in  diagnostic  impressions  regarding  autism 
when  observers  are  evaluating  live  versus  telemedicine?,  and  is  there  a  difference  in 
satisfaction  between  onsite  and  telemedicine  evaluations  of  autism?. 

In  general,  results  from  this  pilot  study  are  favorable  in  that  there  was  little  difference  in 
the  reliability,  utilizing  percent  agreement,  when  the  autism  measures  were  scored  live 
or  over  telemedicine.  For  the  ADOS,  a  significant  difference  in  mean  percent 
agreement  between  conditions  was  only  found  for  one  item  (A7),  which  was  related  to 
socially  directed  pointing.  All  other  items  were  similar  in  agreement  between  the  pairs  of 
researchers,  regardless  of  condition.  These  findings  suggest  clinicians  were  able  to 
score  the  play  based  autism  measure,  ADOS,  equally  well  when  viewing  the  child  live 
or  over  telemedicine,  providing  beginning  evidence  this  tool  can  be  effective  when  used 
over  telemedicine.  Similar  results  were  found  for  the  ADI-R,  which  is  the  parent 
interview  used  to  evaluate  characteristics  of  an  autism  spectrum  disorder  over  the 
child’s  development.  Again,  only  one  item  on  the  ADI-R  (unusual  sensory  differences) 
showed  a  significant  difference  in  mean  percent  agreement  between  conditions,  which 
was  related  to  unusual  sensory  differences.  These  data  also  provide  preliminary 
evidence  that  the  ADI-R  can  also  be  used  over  telemedicine  equally  well  as  in  a 
traditional  clinic  setting  (live).  Although  we  found  little  difference  in  percent  agreement 
between  conditions,  it  is  important  to  note  that  overall  percent  agreement  (regardless  of 
condition)  was  variable  and  did  not  consistently  meet  the  goal  of  85%  agreement  for 
most  items.  Clinicians  did  reach  this  criterion  during  training;  however  criterion  was  only 
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reached  one  time  during  training,  with  percent  agreement  consistently  between  67% 
and  72%  prior  to  beginning  the  study. 

In  addition  to  the  autism  tools,  identification  of  diagnosis  was  also  evaluated.  Clinicians 
agreed  almost  100%  of  the  time  on  the  diagnosis  of  the  child,  regardless  of  the 
condition.  Accuracy  of  diagnosis  was  defined  if  the  diagnosis  following  the  assessment 
matched  the  previous  diagnosis  the  child  received.  Accuracy  was  83%  and  86%  for  live 
and  telemedicine  respectively,  with  clinicians  correctly  identifying  the  diagnosis  in  19  of 
the  21  cases.  It  is  important  to  note  that  clinicians  agreed  100%  of  the  diagnosis  of  the 
child  in  both  of  the  cases  where  the  diagnosis  did  not  match  the  previous  diagnosis. 
Although  inclusion  criteria  attempted  to  include  children  with  an  appropriate  diagnosis,  it 
is  possible  that  the  child’s  diagnosis  had  changed  from  the  initial  visit  and  the  study 
visit,  suggesting  the  research  was  100%  accurate  in  identification  of  diagnosis.  All  of 
the  children  had  been  referred  because  of  red  flags  for  autism  so  it  is  understandable 
the  diagnosis  may  have  changed  in  some  cases. 

Lastly,  we  hoped  families  would  be  equally  satisfied  with  their  diagnostic  experience 
regardless  of  being  conducted  live  or  over  telemedicine.  Results  from  the  satisfaction 
survey  showed  high  levels  of  satisfaction  for  all  families  with  no  significant  differences  in 
satisfaction  between  conditions.  Although  a  moderate  effect  size  was  noted  in  favor  of 
the  live  setting  for  satisfaction  across  all  7  items  of  the  survey,  this  may  not  be  a 
clinically  meaningful  difference  with  ratings  of  6.57  and  6.23  out  of  7  in  the  live  and 
telemedicine  conditions  respectively  suggesting  families  were  highly  satisfied  with  their 
experience,  regardless  of  setting. 

The  pilot  data  will  give  us  information  for  a  prospective  study  with  sufficient  power  to 
determine  if  there  is  a  difference  between  telemedicine  and  live  visits.  Also  the  fact  that 
children  were  already  diagnosed  may  have  affected  parents’  perception  of  the 
telemedicine  diagnostic  process. 

Results  from  this  pilot  study  provide  preliminary  evidence  that  autism  diagnostic 
assessments  can  be  conducted  equally  well  over  telemedicine  as  compared  to  the 
traditional  clinic  setting  (live).  Clinicians  were  able  to  score  measures  as  well  for  almost 
all  items  and  families  reported  high  levels  of  satisfaction  in  both  conditions.  With  the 
increasing  demand  for  trained  professionals  to  conduct  diagnostic  assessments  for 
autism,  the  use  of  telemedicine  provides  a  viable  option  for  families  in  rural  areas  that 
may  not  have  access  to  specialty  care  and  offers  a  considerable  cost  savings  to  these 
families  as  well.  Families  often  travel  long  distances  for  similar  evaluations  resulting  in 
travel  expenses  (gas,  hotel,  food)  as  well  as  lost  work  time.  Telemedicine  provides 
families  with  a  more  cost  effective  means  to  access  needed  medical  care. 

While  results  from  this  pilot  study  are  important  and  suggest  telemedicine  as  a  viable 
option  for  autism  diagnostic  evaluations,  it  is  important  to  discuss  some  potential 
limitations  to  the  use  of  the  telemedicine  technology  for  this  purpose.  The  use  of 
technology  provides  risk  for  equipment  problems  or  connection  problems  that  would  not 
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occur  in  a  traditional  clinic  setting.  We  did  experience  a  few  problems  connecting  to 
telemedicine  sites,  causing  a  few  families  additional  wait  time  for  technology  problems 
to  be  resolved.  Although  in  our  study  we  were  able  to  successfully  connect  and  use  the 
telemedicine  equipment  for  all  21  participants,  we  did  experience  problems  with  the 
family  being  able  to  see  the  clinicians  over  telemedicine  on  one  occasion.  Although  the 
clinicians  were  able  to  see  and  hear  the  family  and  appropriately  score  the  measures, 
these  types  of  problems  using  technology  could  contribute  to  difficulties  establishing 
rapport  with  families,  frustration  with  communication,  or  other  differences  that  would  not 
be  experienced  in  the  clinic  setting.  Limitations  of  telemedicine  may  also  include 
difficulties  or  differences  seeing  or  hearing  behaviors  in  the  child  during  the 
assessment.  Although  we  only  detected  a  significant  difference  in  scores  for  one  item 
related  to  pointing  in  this  study,  there  is  a  potential  for  difficulties  seeing  or  hearing 
because  of  audio  equipment,  angles  of  cameras,  or  placement  of  people  in  the  room. 
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Appendix  1.  Telemedicine  Satisfaction  Survey 

Telemedicine  Consultation  Evaluation  -  Revised 

Strongly  Strongly 

Disagree  Neutral  Agree 

1 .  I  felt  I  could  talk  12  3  4  5  6  7 

about  anything  with 

the  clinicians 

2.  The  clinicians  cared  1  2  3  4  5  6  7 

about  me  as  a  person 

3.  The  clinicians  knew  1  2  3  4  5  6  7 

what  they  were  doing 

4.  I  was  embarrassed  or  12  3  4  5  6  7 

self-conscious  during 

the  session 

5.  I  had  difficulty  healing  12  3  4  5  6  7 

the  clinicians 

6.  I  had  no  trouble  seeing  1  2  3  4  5  6  7 

the  clinicians 

7.  Overall,  I  was  very  1  2  3  4  5  6  7 

satisfied  with  today’ s 

visit 

If  your  session  was  conducted  by  evaluators  asking  you  questions  and  instructing  you  how  to 


interact  with  your  child  over  the  TV,  please  answer 

Strongly 

Disagree 

the  following  questions. 

Neutral 

Strongly 

Agree 

8.  During  the  evaluation, 

I  was  nervous  about  the 
TV  equipment 

1  2 

3  4  5  6 

7 

9.  During  the  evaluation, 
my  child  seemed 
distracted  by  the  TV 
equipment 

1  2 

3  4  5  6 

7 

10.  The  delay  in  hearing  or 

1  2 

3  4  5  6 

7 

talking  to  the  clinician 
was  too  long 


1 1 .  There  were  no 
significant  glitches  with 
the  TV  or  connection 

12.  The  care  I  received 
today  was  as  good  as 
my  in-person  visits 

13.  Telemedicine  makes  it 
easier  to  get  medical 


Supporting  Data 

Table  1 .  Mean  percent  agreement  on  ADOS  items  by  condition. 
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Live  Percent 
Agree 

TM  Percent 

Agree 

Difference  TM  -  Live 

ADOSal:  language  level 

71% 

71% 

No  difference 

ADOSa2:  verbalizations 

52% 

65% 

13  % 

ADOSa3:  intonation 

67% 

65% 

-2% 

ADOSa4:  immediate  echolalia 

67% 

65% 

-2  % 

ADOSa5:  stereotyped  language 

71% 

75% 

4% 

ADOSa6:  contact  gestures 

86% 

80% 

-6% 

ADOSa7:  pointing* 

86% 

30% 

-  56  %* 

ADOSa8:  gestures 

43% 

65% 

22% 

ADOSbl:  eye  contact 

86% 

90% 

4% 

ADOSb2:  social  smile 

71% 

70% 

-  1  % 

ADOSb3:  facial  expressions 

62% 

80% 

18  % 

ADOSb4:  gaze  &  social 
overture 

62% 

50% 

-  12% 

ADOSb5:  shared  enjoyment 

52% 

70% 

18  % 

ADOSb6:  response  to  name 

67% 

75% 

8  % 

ADOSb7:  requesting 

57% 

50% 

-7% 

ADOSb8:  giving 

67% 

55% 

-12  % 

ADOSb9:  showing 

67% 

45% 

-22% 

ADOSblO:  initiation  of  JA 

71% 

60% 

-11% 

ADOSbl  1:  response  to  JA 

91% 

90% 

-  1  % 

ADOSbl2:  quality  of  overtures 

62% 

65% 

3  % 

ADOSc  l:  functional  play 

57% 

70% 

13% 

ADOSc2: 

imagination/creativity 

62% 

50% 

-12% 

ADOSdl:  unusual  sensory 

71% 

90% 

19  % 

ADOSd2:  hand/finger 
behaviors 

81% 

90% 

9% 

ADOSd3:  self  injurious 
behavior 

100% 

100% 

No  difference 

ADOSd4: 

repetitive/ stereotyped 

67% 

80% 

13  % 

ADOSel:  overactivity 

71% 

40% 

-31  % 

ADOSe2:  disruptive  behaviors 

76% 

80% 

4% 

ADOSe3:  anxiety 

91% 

85% 

-6% 

Total  agreement  for  ADOS 

70% 

69% 

1% 

JA  =  Joint  attention,  *  denotes  statistically  significant  difference  between  conditions. 
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Table  2.  Mean  percent  agreement  on  ADI  items  by  condition. 


Live  Percent 
Agree 

TM  Percent 

Agree 

Difference  TM  -  Live 

ADI50:  eye  gaze 

86% 

91% 

5  % 

ADI5 1 :  social  smiling 

86% 

91% 

5  % 

ADI57 :  facial  expressions 

71% 

91% 

20% 

ADI49:  imaginative  play  w/  peers 

81% 

86% 

5  % 

ADI62:  interest  in  children 

71% 

81% 

10% 

ADI63:  response  to  children 

86% 

86% 

No  difference 

ADI64:  group  play  w/  peers 

91% 

85% 

-6% 

ADI52:  showing/directing 

86% 

100% 

14% 

ADI53:  offering  to  share 

95% 

91% 

-4% 

ADI54:  shared  enjoyment 

81% 

86% 

5  % 

ADI3 1 :  contact  gestures 

81% 

91% 

10% 

ADI55:  offering  comfort 

95% 

95% 

No  difference 

ADI56:  quality  of  social  overtures 

71% 

91% 

20% 

ADI58:  inappropriate  facial 

86% 

81% 

-5% 

ADI59:  social  response 

76% 

91% 

15% 

ADI42:  pointing  to  express 
interest 

76% 

76% 

No  difference 

ADI43:  nodding 

100% 

91% 

-9  % 

ADI44:  head  shaking 

95% 

95% 

No  difference 

ADI45:  gestures 

91% 

86% 

-5  % 

ADI47 :  spontaneous  imitation 

81% 

81% 

No  difference 

ADI48:  imaginative  play 

86% 

91% 

5  % 

ADI61:  imitative  social  play 

71% 

81% 

10% 

ADI34:  social  chat 

76% 

91% 

15% 

ADI35:  reciprocal  conversation 

81% 

76% 

-5% 

ADI33:  stereotyped 
lang/echolalia 

76% 

86% 

10% 

ADI36:  inappropriate  questions 

91% 

91% 

No  difference 

ADI37 :  pronominal  reversal 

76% 

86% 

10% 

ADI38:  neologisms/idiosyncratic 

91% 

86% 

-5% 

ADI67 :  unusual  preoccupations 

91% 

76% 

-15% 

ADI68:  circumscribed  interests 

86% 

71% 

-15% 

ADI39:  verbal  rituals 

86% 

86% 

No  difference 

ADI70:  compulsions  /  rituals 

95% 

91% 

-4% 

ADI77:  hand  or  finger 
mannerisms 

91% 

91% 

No  difference 

ADI78:  complex  body 
movements 

81% 

86% 

5  % 

ADI69:  interest  in  parts  of  objects 

76% 

91% 

15% 

ADI7 1 :  unusual  sensory  interests* 

67% 

95% 

28%* 

Total  Agreement  on  all  items 

83% 

87% 

-4% 

denotes  statistically  significant  difference  between  conditions. 
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Table  3.  Mean  satisfaction  ratings  by  condition. 


Condition 

N 

Mean 

Std.  Deviation 

t-test 

Effect  Size  (d) 

Evaluationl 

Live 

11 

6.82 

.41 

t(19)=.89 

.38 

Comfort  w/  talking  to  clinicians 

ITV 

10 

6.30 

1.89 

p=.39 

Evaluation2 

Live 

11 

6.73 

.47 

t(19)=.91 

.39 

Felt  clinicians  cared 

ITV 

10 

6.20 

1.87 

CO 

CO 

if 

CL 

Evaluations 

Live 

11 

6.91 

.30 

t(8.30)=.86 

.41 

Competent  clinicians 

ITV 

9 

6.33 

2.00 

C\J 

if 

CL 

Evaluation  4 

Live 

11 

6.00 

1.90 

t(19)=.95 

-.10 

Felt  embarrassed 

ITV 

10 

6.20 

1.93 

p=.36 

Evaluation  5 

Live 

11 

6.91 

.30 

t(19)=.70 

.30 

Difficulty  hearing 

ITV 

10 

6.70 

.95 

o 

LO 

if 

CL 

Evaluation  6 

Live 

11 

5.82 

2.40 

t(19)=.81 

.35 

No  trouble  seeing 

ITV 

10 

4.90 

2.77 

CO 

"sf 

if 

CL 

Evaluation? 

Live 

11 

6.82 

.41 

t(10)=  -1 .49 

-.62 

Overall  satisfaction 

ITV 

10 

7.00 

.00 

P=17 

Mean  Satisfaction  ltemsl-7 

Live 

11 

6.57 

.49 

t(19)=1 .16 

.50 

ITV 

10 

6.23 

.82 

p=.26 

Evaluation  questions  #4  and  #5  were  reversed  scored  to  create  high  scores  =  high 
satisfaction 


